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Sir^ 

I, MICHAEL D. DOYLE, hereby declare chat : 

1. I em a co-'inventor of the subject matter disclosed 
and claimed in U.S. Patent Application No. os/324,443- 

2. The subject patter claimed in the above patent 
application was reduced to practice in this country prior to 
October, 1934, the publication date of the cited reference 
entitled "Mosaic and the World-Wide Web" by Venter. 

3. The reduction to practice of the claimed invention, 
is evidenced by ATTACHMENT A, which is a transcript of the audio 
portion and still photographs of a video tape presented to ar. 
audience of scientists prior to October, 195-1. 

4. As stated in ATTACHMENT A, starting at the bottorr, 
of page 2, interface and control software had been developed that 
allows the embedding of a visualization application within a 
Mosaic document. As is apparent from the photographs, the object 
is displayed and processed within the browser-controlled window. 
The visualization application is external to the hypermedia 
dc»cuTO*nt displayed by the browser. 
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I further declare that all statements made herein of -n> 
own knowledge are true and that all statements made on 
informarion and belief are believed to be true; and further than 
these statements were made with the knowledge that willful fal«e 
statement© and the like so made are punishable by fine or 
imprisonment , or both, under Section 1001 of Title 18 of the 
United States Code, and that such willful false statements may 
jeopardize the validity of the application or any patent issuing 
thereon. 

Dated- ^ 7 _ . 1997. 




MICHAEL D, DO' 
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M.D. Doyle, et. aL, The Virtual Embryo: VR Applications in Human 
Developmental Anatomy, presented at * Medicine Meets Virtual Reality II, 
Interactive Technology and Healthcare: Visionary Applications for Simulation, 
Visualization & Robotics," sponsored by the UCSD School of Medicine and the 
Advanced Projects Research Agency, San Diego, CA, January 27, 1994. 

Video Presentation Transcript: 

This is a status report of some of the work that's been accomplished during the 
first years of the Visible Embryo Project. 

One of our first tasks was to develop some volume visualization software that we 
could use for imaging and analysis of the embryo reconstructions that we 
planned to create during the full term of the embryo project. One thing that was 
an absolute requirement was that this software be able to distribute its 
computational load across a network of graphics computers that weren't 
necessarily all in the same place. Basically we wanted to be able to have 
computers that could be all over the country connected by high -speed 
networking, able to contribute to a computation of three-dimensional datasets. 




What you see here is a package called "VIS/' which was developed in our 
group, for three-dimensional volume visualization. This is a very portable 
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package that has been generated using as generic code as possible, although 
this particular image that you see here is running on a Silicon Graphics Reality 
Engine II, which is optimized for volume visualization. We need to use that kind 
of high-speed optimization to accomplish the real-time interactivity that we need 
to accomplish for this project. And as you see, as this rotates, and it starts 
rotating faster and faster, only a very powerful graphics - basically a graphics 
supercomputer - can accomplish this much computation on a three dimensional 
dataset. One of our goals is to allow anybody on the Internet with a very low- 
level access workstation to accomplish this kind of interactivity through their 
network connection, and the way that we do that is through a client-server 
architecture where we have a very powerful server computer accessed by a very 
low-end client machine. 

We decided early on to use NCSA's Mosaic program and the World Wide Web 
to integrate access to this. system. One problem with Mosaic is, as it exists 
today, is that the images within Mosaic are typically static or passive-playback 
images. 




What you see here is an enhancement that we've created to the Mosaic 
interface and control software that allows the embedding of a dynamic real-time 
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visualization application within a Mosaic document. You see a head, a volume 
MRI model of a human head, that's being rotated in real-time interactively by the 
viewer. You see a little panel to the right which is, it's a control panel that's 
popped up - it's really external to Mosaic itself but it can talk to the internal 
control programs that drive the Mosaic client - and allow you to interactively 
control the display from within Mosaic. This is actually controlling the volume 
visualization software that you just saw a few minutes ago. 
By moving around the controls in the control panel we can do things like rotate, 
we can control slices through the dataset, and so on. As you can see, there's 
rotation in x T y, and z planes. We can also compute arbitrary oblique sections 
through the data and look at the internal anatomy. Here we can see the brain of 
this individual; we can rotate and view that section of the dataset from a variety 
of vantage points. There are zooming capabilities that allow us to zoom up on 
the data or zoom back to look at things in more or less detail, and you can see 
here that once zoomed, we're moving our cutting plane down through the 
dataset and looking at more and more inferior levels. 




Normally graphics in Mosaic are static, as I said, but this embedding of graphic 
applications within Mosaic, is really going to form the basis of how we integrate 
information access through the Visible Embryo Project. What you are about to 
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see is our prototype system of a Visible Embryo Mosaic document that has 
embedded realtime visualizations within it. 

We just loaded the Visible Embryo Mosaic page, the system is about to scroll 
down this page - it is an abstract about the Visible Embryo Project - and then 
we see a window. It looks like a static image just like is normally found within the 
World Wide Web databases that you can access today through Mosaic, but you 
can see that suddenly, by moving controls on the control panel, we can zoom in 
and see that this is a reconstruction of a seven week old human embryo. This is 
a reconstruction from approximately 2900 serial cross sections of an embryo 
sectioned about in the 1930s. It's part of the Carnegie Collection of Human 
Embryology. 
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We're looking at this in volume visualization mode, we can rotate the embryo 
around, we can see internal structures, neurological structures; just in the lower 
abdomen area, we can see the liver, the arms are very evident - so we are 
actually looking through the dataset. We can also slice through this dataset 
obliquely, and look at the internal anatomy that way as well. We can load a 
volume visualization table that allows us to interactively enter tissue 
characteristic numbers that control the translucency, transparency, or opacity of 
various ranges of voxal intensity. And what we've done now is we've made the 
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exterior of the embryo a little more opaque so that as we rotate around with an 
oblique cutting plane we can see the difference between the cutting plane and 
the exterior of the embryo a little better. So now we're looking, slicing - we've 
rotated the embryo to an inferior view and we're looking up at the embryo from 
below. And we can see, we've just gone through the heart region, we're going 
through the liver, we're moving inferiorly and we start to get to an area where we 
can see the herniated gut. 

Now the real key to all of this is that these are embedded visualizations. We're 
actually creating documents that are - I guess you'd call them currently 
compound documents where you have the traditional type of information, but 
you've also got t within that document, links to the raw data rather than just 
pictures generated from data. This allows you to tie together representations of 
data with the actual data themselves as well as with notes and different kinds of 
descriptive textual information based on that data. 



Our basic objective here is really to create what we're calling a national meta- 
center which is going to be a computational resource for the entire nation that 
allows people interested in many different areas, including developmental 
biology, also multi-dimensional imaging and high-speed networking, and 
parallel computing. Ail these people can access this database. And the parallel 
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nature of the computation that's taking place is invisible to the user. They log in 
through a Mosaic window, and that window is giving them very high-performance 
control of interactive visualizations of datasets. By scrolling the window, we can 
see that this is actually embedded within the Mosaic document. 

During a recent demonstration of this technology at the corporate briefing center 
at Silicon Graphics Corporation in Mountain View, California, I discussed some 
of the implications of this technology for researchers of human genetics. 



"We're also looking at using these models as a basis for creating three- 
dimensional maps of gene expression, which is a way to correlate the findings of 
the Human Genome project within a context that everyone uses. It sort of sets 
up a standard space within which everybody can report their findings, so that 
you can finally have some way of comparing studies that happen in different 
laboratories. 




If you're studying the three-dimensional distribution of gene expression of a 
gene relating to heart development, what you do now is you have a little 
fluorescent marker that glows under an ultraviolet light, and you use confocal 
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microscopy to develop a three-dimensional model of it, and then you say, well, 
it's on here and it's on there and it's on there and you try to describe it in 
anatomical terms but it is a qualitative description, right?. 

But here there would be a standard anatomy space that people could use to 
describe their findings so that they could, rather than say, yeah, we saw five 
different studies that said that this was expressed at the bifurcation of the aorta 
with the Common Carotid artery - you don't have to do in terms of verbal 
descriptions, you can do it in terms of a true measurable Cartesian coordinate 
system. 



If you take your current version of Mosaic, the kind that is accessible for free 
through the Internet today, and log onto our home page of the Visible Embryo 
Project, what you'll see is a series of multi-media documents that basically give 
q you information about the status of the Visible Embryo Project and the status of 
fg our current proposal development efforts. You'll be able to load MPEG movies of 
kj visualizations of human embryos, as you can see here. 
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One thing you should keep in mind is that this little MPEG movie is just a canned 
movie, it's not interactive. Once you hit Play it just goes and it plays and then it 
goes away, but you can't stop it and interact with it, and rotate that embryo, for 
instance, to different vantage points. 

Also available is an image that shows some of the early work t some screen 
shots representing the volume visualization tool current as of about last summer. 
We've come much farther and in fact a lot of the video that you saw earlier in 
this presentation shows you the current status of this volume visualization 
package. Last summer, that imaging package was separate from Mosaic, as you 
see it's off to the right, and Mosaic could just call it but couldn't actually embed 
visualizations. Now, everything is tied together into a single multi-media 
document. 



You'll also see articles that relate to the Visible Embryo Project This project has 
been going on for several years now, mostly on the coattails of other research 
projects, collaborators funding it wherever they could find the money. But 
already a significant body of literature is starting to be built up around this 
project. 
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Of course, in the very near future, you'll be able to log on through an enhanced 
version of Mosaic We're working together with the National Center for 
Supercomputing Applications on enhancing the standard release of Mosaic 
to allow these capabilities. And you'll be able to access interactive dynamic 
visualizations that are being served by a network of high-end supercomputers 
across the country. Even if you're only accessing the system with a machine like 
a Macintosh or PC, you'll still be able to access the power of these 
supercomputers from your own location. 




What I've attempted to demonstrate in the last several minutes in this 
presentation is that there's been a considerable amount of work already done in 
this project that we call the Visible Embryo Project. Many collaborators across 
the country have worked together to create a set of enabling technologies to 
allow this project to accomplish its goals. The Visible Embryo Project represents 
an effort to serve the needs of both the biology community as well as the 
information science community, in that we are attempting through current 
applications in information technology to break through barriers that have 
prevented biological researchers from asking and answering questions about the 
most fundamental mechanisms of human growth and development. We're also 
creating a technology development testbed for the information sciences that will 
allow researchers to push the envelope, so to speak, of information technologies 
to their very limits. 
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SPEAKER CONSENT FDR AUDIO/Vi nFQ RECORDINGS 

We would like to audio/video tape your presentation at Medicine Me«ts virtual Reality ii: 
Interactive Technology and Healthcare.January 27-30. 1994. 

Audio and video tapes will be available for immediate distribution to attendees and will be 
marketed and sold after the conference. Your colleagues will be able to benefit from your 
remarks by listening to cassettes and viewing tapes whether or not they were in attendance. 
The tapes will be copyrighted and marketed under Title 17 of the U.S. Code or other law as may 
be enacted and all rights to royalties, if any, in conjunction with cassette sales shall hereby be 
assigned to Medicine Meets Virtual Reality. This agreement does not preclude the publication 
by you of your paper, speech or comments at any time and in any format. 

Please cooperate with the audio/visual taping staff to obtain the best possible recordings by 
using the microphone at all times and repeating the questions that are asked by persons not 
near a microphone. 

Sign this form below and return it to us at your earliest convenience. 
TSank You. 



ifjsreby give permission to Medicine Meets Virtual Reality to record and distribute audio 
aW video tapes containing my presentation as outlined above. 




SJgnature_ 

F§ase print namp /Jtc-Ai* / ^. 7)oy^<. 



flank you very much for your cooperation. Remember to pick up the complimentary audiotape 
dfyour presentation before leaving the meeting. 



Please return this completed form to: 
MEDICINE MEETS VIRTUAL REALITY 
P.O. Box 23220, San Diego. CA 92193 
For further information call 619/751-8841, 
or Fax 619/751-8842. 
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UNIVERSITY OF CALIFORNIA, SAN DIEGO 
OFFICE OF CONTINUING MEDICAL EDUCATION 
DISCLOSURE STATEMENT 

It is ihe policy of the UCSD Office of Continuing Medical Education to insure balance, independence, objectivity, and 
scientific rigor in all its individually sponsored or jointly sponsored educational programs. All faculty participating in any 
UCSD sponsored programs are expected to disdose to the program audience anv real or apparent conffia(s) of Interest 
that may have a direct bearing on the subject matter of the continuing education program. This pertains to relationships 
wrth pharmaceutical companies, biomedical device manufacturers, or other corporations whose products or sen/ices are 
realated to the subject matter of the presenatation topic. The intent of this policy is not to prevent a speaker with a 
potential conflict of interest from making a presentation. It is merely intended that any potential conflict should be 
identified openly so that the listeners may form their own Judgments about the presentation with the full disclosure of the 
f^cts. It remains for ihe audience to determine whether the speaker's outside interests may reflect a possible bias in either 
the exposition or the conclusions presented. 

MEDICINE MEETS VIRTUAL REALITY II 

INTERACTIVE TECHNOLOGY & HEALTHCARE 

January 27-30* 1994 



BQ PRESENTER'S NAME;_ 




(please print or type) 
I hav4So actual or potential conflict of interest in relation totfjifi 

Signature 



I havC| financial interest/arrangement or affiliation with one or more organizations that could be percaiveod as a real or 
appateht conflict of Interest in the context of the subject of ibis presentation. 

AfTTlja^n/Flnancigl Intgrggt Name of Organization^ 

Grantftassarch Support 

Consultant . 

Speakers 1 Bureau 

Major Stock Shareholder 

Other Financial or Material Support ' . 



Signature (date) 
Your cooperation in complying with these guidelines is appreciated. 



Please return this form as soon as possible to: Medicine Meets Virtual Reality II, P.O. Box 23220, San Diego, CA 92193 
Fax: 619-751-8342: Phone: 619-751-8841 
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